Corneal reflected image identifying device and gaze detection device

ABSTRACT

A corneal reflected image identifying device and gaze detection device enabling detection of a gaze with a high precision even when a person being monitored is wearing an object covering the eyes, in particular a corneal reflected image identifying device comprising a light projecting part  110  for projecting light from two light sources  110   a,    110   b  which are spaced apart from each other toward an eye, a camera  120  for capturing an image of the eye on which light is projected, a distance calculating part  152   d  for using the captured image to calculate a distance between two reflected images for each combination of two reflected images among a plurality of reflected images obtained by light projected from the light sources  110   a,    110   b  being reflected in a vicinity of the eye, and a corneal reflected image identifying part  152   g  for excluding a reflected image of light projected from the light sources  110   a,    110   b  being reflected by an object covering the eye based on predetermined features and identifying a corneal reflected image to be used for gaze detection, wherein the predetermined features include the distance which the distance calculating part  152   d  has calculated.

FIELD

The present invention relates to a corneal reflected image identifyingdevice and gaze detection device.

BACKGROUND

In the past, as described in Japanese Unexamined Patent Publication No.2018-28728, there has been the art of using a plurality of synchronizedcameras to generate a plurality of corneal reflected images and usingthe distances and angles of pairs of the same to select a correctcorneal reflected image at each camera.

SUMMARY Technical Problem

If projecting light to an eye and using a reflected image of the lightat the cornea to detect a gaze, if the person being monitored is wearingeyeglasses, the image of the projected light reflected at the lens,frame, etc. of the eyeglasses might be mistakenly detected as thereflected image of the cornea. Further, if such mistaken detectionoccurs, there is the problem of a drop in the precision of detection ofthe gaze.

The art described in the above patent literature uses a plurality ofsynchronized cameras to generate a plurality of corneal reflectedimages, but there is one corneal reflected image corresponding to onecamera. The true corneal reflected images of the cameras are selected toexpand the range of gaze detection. For this reason, the art describedin the patent literature is art predicated on the presence of a cornealreflected image of reflection at the cornea. When detecting the gaze ofan eyeglass wearer, the problem remains of an image reflected at thelens, frame, etc. of the eyeglass being mistakenly detected as areflected light of the cornea.

Therefore, the present invention has as its object the provision of acorneal reflected image identifying device and gaze detection deviceenabling detection of a gaze with a high precision even if a personbeing monitored is wearing an object covering the eyes.

Solution to Problem

According to one embodiment, a corneal reflected image identifyingdevice is provided.

This corneal reflected image identifying device is comprised of a lightprojecting part for projecting light from two light sources which arespaced apart from each other toward an eye, an imaging part forcapturing an image of the eye on which light is projected, a distancecalculating part for using the captured image to calculate a distancebetween two reflected images for each combination of two reflectedimages among a plurality of reflected images obtained by light projectedfrom the light sources being reflected in a vicinity of the eye, and acorneal reflected image identifying part for excluding a reflected imageof light projected from a light source being reflected by an objectcovering the eye based on predetermined features and identifying acorneal reflected image to be used for gaze detection, wherein thepredetermined features include the distance which the distancecalculating part has calculated.

In this corneal reflected image identifying device, the cornealreflected image identifying part preferably excludes a combination oftwo reflected images where the distance exceeds a first threshold valueand identifies one reflected image included in a combination of tworeflected images where the distance is the first threshold value or lessas a corneal reflected image to be used for gaze detection.

In this corneal reflected image identifying device, preferably an anglecalculating part for calculating an angle of a line connecting tworeflected images with a line connecting the two light sources whenviewed from the imaging direction of the imaging part for eachcombination of two reflected images is provided, the predeterminedfeatures include the angle which the angle calculating part hascalculated, and the corneal reflected image identifying part excludes acombinations of two reflected images where the angle exceeds a secondthreshold value and identifies one reflected image included in acombination of two reflected images where the angle is the secondthreshold value or less as the corneal reflected image to be used forgaze detection.

In this corneal reflected image identifying device, preferably an areacalculating part for calculating an area of a reflected image for eachcombination of two reflected images is provided, the predeterminedfeatures include the area which the area calculating part hascalculated, and the corneal reflected image identifying part excludes acombination of two reflected images where an area exceeds a thirdthreshold value and identifies one reflected image included in acombination of two reflected images where the areas are the thirdthreshold value or less as the corneal reflected image to be used forgaze detection.

In this corneal reflected image identifying device, preferably athreshold value correcting part for correcting the first threshold valueto a larger value the longer the distance from the light sources to theface is provided.

In this corneal reflected image identifying device, preferably athreshold value correcting part for correcting the third threshold valueto a larger value the longer a distance from the light sources to theface is provided.

According to one embodiment, a gaze detection device is provided. Thisgaze detection device is provided with a gaze detecting part fordetecting a gaze by a corneal reflection method based on a cornealreflected image identified by the above corneal reflected imageidentifying device.

Advantageous Effects of Invention

According to the present invention, the effect is exhibited that itbecomes possible to provide a corneal reflected image identifying deviceand gaze detection device enabling detection of a gaze with a highprecision even if a person being monitored is wearing an object coveringthe eyes.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic view of the configuration of an occupantassistance system in which a corneal reflected image identifying deviceand gaze detection device according to an embodiment are mounted.

FIG. 2 is a schematic view showing a camera unit comprised of a lightprojecting part and a camera integrally joined.

FIG. 3 is a schematic view showing a camera unit comprised of a lightprojecting part and a camera integrally joined.

FIG. 4 is a view showing an image, in which an eye of a driver isappeared, generated by capturing an image of the eye of a driver wearingeyeglasses by a camera unit shown in FIG. 2 or FIG. 3 .

FIG. 5 is a schematic view showing function blocks of a processor of acontrol device.

FIG. 6 is a schematic view showing a state of a distance calculatingpart calculating a distance between two candidates of reflected imagesin the example of FIG. 4 .

FIG. 7 is a schematic view showing a state of an angle calculating partcalculating an angle formed by a line connecting two candidates ofreflected images with the horizontal.

FIG. 8 is a view showing enlarged a positional relationship between thetwo images of a corneal reflected image and eyeglass reflected imageshown in FIG. 7 .

FIG. 9 is a flow chart showing processing which a processor of a controldevice performs for each predetermined control cycle.

DESCRIPTION OF EMBODIMENTS

FIG. 1 is a schematic view of the configuration of an occupantassistance system 100 in which a corneal reflected image identifyingdevice and gaze detection device are mounted according to oneembodiment. The occupant assistance system 100 is mounted in anautomobile or other vehicle and detects a gaze of the driver or otheroccupant. Note that, in the following explanation, the case where theoccupant is the driver will be explained as an example. The occupantassistance system 100 has a light projecting part 110 for projectinglight toward a face of the driver, a camera 120 for capturing the faceof the driver and generating an image, a display device 130 fordisplaying a warning to the driver etc., a speaker 140 for generating awarning etc. to the driver by voice, and a control device (ECU) 150. Thelight projecting part 110, camera 120, display device 130, speaker 140,and control device 150 are connected to be able to communicate throughan internal vehicle network based on the Controller Area Network (CAN)standard.

The light projecting part 110 has two light sources which are spacedapart from each other and projects light from the two light sourcestoward an eye of the driver. The two light sources are, for example,comprised of near infrared light sources (near infrared LEDs) emittingnear infrared light.

The camera 120 is one mode of the imaging part and has a 2D detectorcomprised of an array of photoelectric conversion elements havingsensitivity to visible light or near infrared light such as CCDs orC-MOS's and an image-forming optical system for forming an image of anarea to be captured on that 2D detector. The camera 120 is provided on adashboard, steering column, the vicinity of the front glass, etc. at theinside of the vehicle facing the assumed position of the driver andcaptures the face of the driver. The camera 120 captures the eye of thedriver to which the light projecting part 110 projected light everypredetermined imaging cycle (for example, 1/30 second to 1/10 second)and generates an image in which the eye of the driver appears. Notethat, the image obtained by the camera 120 is preferably a color image.Further, the camera 120 may be configured by a stereo camera and may beconfigured by obtaining the distance from the parallax of the left andright images to the different structures on the image. The camera 120outputs the generated image every time generating an image through theinternal vehicle network to the control device 150. Note that, asexplained later, the camera 120 may be configured integrally with thelight projecting part 110.

The display device 130 is, for example, configured from a liquid crystaldisplay device (LCD) etc. and is provided at the instrument panel or thevicinity of the dashboard etc. and displays a warning to the driver inaccordance with need. The speaker 140 is provided in the vicinity of thedashboard etc. and in accordance with need issues a warning by voice tothe driver.

The control device 150 is a constituent element controlling the occupantassistance system 100 as a whole. The corneal reflected imageidentifying device and gaze detection device according to the presentembodiment is configured by the control device 150, light projectingpart 110, and camera 120. The control device 150 has a processor 152,memory 154, and communication interface (I/F) 156. The processor 152further has one or more CPUs (central processing units) and theirperipheral circuits. The processor 152 may further have anotherprocessing circuit such as a logical operation unit, numerical operationunit, or graphic processing unit. The memory 154, for example, has avolatile semiconductor memory and nonvolatile semiconductor memory andstores programs, data, etc. relating to processing according to thepresent embodiment. The communication interface 156 has an interfacecircuit for connecting the control device 150 to the internal vehiclenetwork.

The control device 150 detects the gaze of the driver. Note that, in thepresent embodiment, “detecting the gaze” means detecting the directionof the gaze (angle of gaze). As the technique for calculating the gazeangle, the technique of utilizing a reflected image of a near infraredlight source at the corneal surface (Purkinje image) and calculating thegaze angle from the distance between the center of the pupil and thePurkinje image, the technique of using the center of curvature of thecornea obtained from the Purkinje image and calculating the lineconnecting the center of curvature of the cornea and the center of thepupil as the gaze angle, etc. are known. In the present embodiment,these techniques use the corneal reflected image (below, also referredto as the “corneal reflection method”) to detect the gaze of the driver.

When detecting the gaze of the driver, if the driver is wearingeyeglasses, the near infrared light which the light projecting part 110projects is reflected by the frame or lens etc. of the eyeglasses. Thereis the possibility of the reflected image of the eyeglasses caused bythis being mistakenly detected as reflected light of the pupil orcornea. If such mistaken detection occurs, the precision of gazedetection falls. In particular, in the case of a single-lens camera,this cannot be handled by interpolation by increasing the number ofcameras or geometric techniques, so it is difficult to suitably dealwith mistaken detection due to a reflected image of reflection at theeyeglasses and there is a possibility of the precision of gaze detectionfalling.

In the present embodiment, by projecting light toward an eye of a driverfrom a light projecting part 110 having two light sources which arespaced apart from each other, two corneal reflected images appear forone eye. Further, eyeglass reflected images are excluded based on adistance between two points of the reflected images, an angle of a lineconnecting two points of the reflected images, and areas of thereflected images, so eyeglass reflected images are kept from beingmistakenly detected as corneal reflected images. Due to this, even inthe case of an eyeglass wearer, more accurate gaze detection becomespossible. Note that, in the present embodiment, eyeglasses wereillustrated as the object covering the eye, but in the case of wearingsunglasses, a face shield, goggles, or other transparent orsemitransparent object covering the eyes as well, in the same way,mistaken detection due to a reflected image of reflection at theseobjects is suppressed and more accurate gaze detection is realized.

FIG. 2 and FIG. 3 are schematic views showing a camera unit 160comprised of a light projecting part 110 and a camera 120 integrallyjoined. FIG. 2 and FIG. 3 show the appearance of the camera unit 160seen from the direction of the optical axis of an image-forming opticalsystem of the camera 120. The camera unit 160 shown in FIG. 2 has acamera 120, a light source 110 a provided at a left side of the camera120 in the horizontal direction when viewing the camera unit 160 fromthe front face, and a light source 110 b provided at a right side of thecamera 120 in the horizontal direction when viewing the camera unit 160from the front face.

Further, the camera unit 160 shown in FIG. 3 has a camera 120, a lightsource 110 a provided so as to surround the camera 120, and a lightsource 110 b provided at a right side of the camera 120 in thehorizontal direction when viewing the camera unit 160 from the frontsurface. The light source 110 a and light source 110 b shown in FIG. 2and FIG. 3 are respectively comprised of sets of pluralities of pointlight sources. Note that, below, the case where the light projectingpart 110 and the camera 120 are comprised of the camera unit 160 shownin FIG. 2 or FIG. 3 will be explained as an example.

FIG. 4 is a view showing an image 50, in which an eye 10 of a driver isappeared, generated by capturing an image of the eye of a driver wearingeyeglasses by a camera unit 160 shown in FIG. 2 or FIG. 3 . As shown inFIG. 4 , in the image 50, the contours 12 and the pupil 20 of the eye 10appear. At the vicinity of the pupil 20, the corneal reflected image 30a of the light of the light source 110 a reflected at the cornea and thecorneal reflected image 30 b of the light of the light source 110 breflected at the cornea appear. Further, at the vicinity of the pupil20, the two images of the eyeglass reflected image 40 and eyeglassreflected image 42 reflected at the eyeglasses appear. Specifically, theeyeglass reflected image 40 and eyeglass reflected image 42 arereflections of light of the light sources 110 a, 110 b reflected at theframe, lenses, etc. of the eyeglasses.

As shown in FIG. 4 , in an image 50 generated by capturing an eye of adriver wearing eyeglasses, eyeglass reflected images 40, 42 appear. Ifthe eyeglass reflected images 40, 42 are mistakenly judged as reflectedimages of the cornea and the eyeglass reflected image 40 or the eyeglassreflected image 42 is utilized to detect the gaze of the driver, thegaze of the driver will be mistakenly detected.

The corneal reflected image 30 a and corneal reflected image 30 b arereflected images of the light source 110 a and light source 110 b at thecornea, so the distance between the corneal reflected image 30 a and thecorneal reflected image 30 b is a value kept within a certain extent inadvance in accordance with the distance between the light source 110 aand light source 110 b.

On the other hand, the eyeglass reflected image 40 or the eyeglassreflected image 42 is an image of light of the light source 110 a or thelight source 110 b reflected at the eyeglasses outside from the cornea,so the distance between the eyeglass reflected image 40 and the eyeglassreflected image 42 or the distance between either of the eyeglassreflected image 40 and the eyeglass reflected image 42 and either of thecorneal reflected image 30 a and the corneal reflected image 30 busually becomes larger than the distance between the corneal reflectedimage 30 a and the corneal reflected image 30 b. Further, the distancebetween the eyeglass reflected image 40 and the eyeglass reflected image42 also usually becomes larger than the distance between the cornealreflected image 30 a and the corneal reflected image 30 b.

In the present embodiment, if extracting two reflected images from theimage 50, finding the distance between the two extracted reflectedimages, and the found distance is a predetermined value or less, it isjudged that the two extracted reflected images are the corneal reflectedimage 30 a and the corneal reflected image 30 b. In this case, thedirection of the gaze of the driver is accurately detected by thecorneal reflection method using the corneal reflected image 30 a orcorneal reflected image 30 b. On the other hand, if extracting tworeflected images from the image 50, finding the distance between the twoextracted reflected images, and the found distance is larger than apredetermined value, it is judged that at least one of the two extractedreflected images is the eyeglass reflected image 40 or the eyeglassreflected image 42.

In this case, mistaken judgment of the gaze is suppressed by notdetecting the gaze based on these two reflected images.

On the other hand, in rare cases the distance between either one of theeyeglass reflected image 40 and eyeglass reflected image 42 and eitherone of the corneal reflected image 30 a and corneal reflected image 30 bwill become the above-mentioned predetermined value or less.

Further, in rare cases the distance between the eyeglass reflected image40 and the eyeglass reflected image 42 will become the above-mentionedpredetermined value or less. For this reason, the eyeglass reflectedimage is excluded based on the angle of a line connecting two points ofthe reflected images.

So long as the reflection surface is not complicated in shape, theorientation of connection of the two light sources 110 a, 110 b and theorientation of connection of the reflected images at the reflectionsurface will become the same. Therefore, the angle of the lineconnecting the two corneal reflected images depends on the angle of theline connecting the two light sources 110 a, 110 b and, when seen fromthe imaging direction, generally matches the angle of the lineconnecting the two light sources 110 a, 110 b. As shown in FIG. 2 andFIG. 3 , in the camera unit 160, the two light sources 110 a and lightsource 110 b are arranged in the horizontal direction, so as shown inFIG. 4 , the corneal reflected image 30 a of the light of the lightsource 110 a reflected at the cornea and the corneal reflected image 30b of the light of the light source 110 b reflected at the cornea appearin a state aligned in the horizontal direction in the image 50.

On the other hand, if the reflected images include an eyeglass reflectedimage, since the eyeglass reflected image is an image of the light ofthe light source 110 a or light source 110 b reflected at the frame orlens of the eyeglasses or other reflection surface of the eyeglasses andthe reflection surface of the eyeglasses faces a random direction, whenviewed from the imaging direction, the angle of the line connecting thereflected image of the eyeglasses and a corneal reflected image will notmatch the angle of the line connecting the two light sources 110 a, 110b.

Similarly, the angle of the line connecting the two reflected images ofthe eyeglasses will not match the angle of the line connecting the twolight sources 110 a, 110 b.

Therefore, in FIG. 4 , the eyeglass reflected image 40 and the eyeglassreflected image 42 basically will never be aligned horizontally and theline connecting the eyeglass reflected image 40 and the eyeglassreflected image 42 will not become horizontal. Similarly, the lineconnecting either one of the eyeglass reflected image 40 and eyeglassreflected image 42 and either one of the corneal reflected image 30 aand corneal reflected image 30 b will not become horizontal.

In the present embodiment, if extracting two reflected images from theimage 50, finding an angle of a line connecting the two extractedreflected images, and the found angle is horizontal, it is judged thatthe two extracted reflected images are the corneal reflected image 30 aand the corneal reflected image 30 b. In this case, the direction of thegaze of the driver is accurately detected by the corneal reflectionmethod using the corneal reflected image 30 a or the corneal reflectedimage 30 b. On the other hand, if extracting two reflected images fromthe image 50, finding an angle of a line connecting the two extractedreflected images, and the found angle is not horizontal, it is judgedthat at least one of the two extracted reflected images is the eyeglassreflected image 40 or the eyeglass reflected image 42. In this case,mistaken judgment of the gaze is suppressed by not detecting the gazebased on these two reflected images.

Furthermore, an area of a corneal reflected image becomes smaller thanan area of a reflected image of the eyeglasses due to the curvature andsize of the reflection surface. The sizes of the light sources 110 a,110 b are set in advance so that the corneal reflected images 30 a, 30 bbecome the minimum limit sizes required for detecting the gaze by thecorneal reflection method. For this reason, the size of the cornealreflected image 30 a of the light of the light source 110 a reflected atthe cornea in the image 50 becomes a size according to the size of thelight source 110 a. Similarly, the size of the corneal reflected image30 b of the light of the light source 110 b reflected at the cornea inthe image 50 becomes a size according to the size of the light source110 b. On the other hand, the eyeglass reflected image 40 or theeyeglass reflected image 42 is an image of the light of the light source110 a or light source 110 b reflected at the frame or lens of theeyeglasses or other reflection surface of the eyeglasses. The size ofthe eyeglass reflected image 40 or eyeglass reflected image 42 becomesgreater than the size of the corneal reflected image 30 a or cornealreflected image 30 b due to the difference between the curvature of thereflection surface of the eyeglasses and the curvature of the cornea orthe difference between the size of the reflection surface of theeyeglasses and the size of the cornea.

In the present embodiment, if extracting a reflected image from theimage 50, finding the area of the extracted reflected image, and thefound area is a predetermined value or less, it is judged that theextracted reflected image is the corneal reflected image 30 a or thecorneal reflected image 30 b. In this case, the direction of the gaze ofthe driver is accurately detected by the corneal reflection method usingthe corneal reflected image 30 a or corneal reflected image 30 b. On theother hand, if extracting a reflected image from the image 50, findingthe area of the extracted reflected image, and the found area is largerthan a predetermined value, it is judged that the extracted reflectedimage is the eyeglass reflected image 40 or the eyeglass reflected image42. In this case, mistaken judgment of the gaze is suppressed by notdetecting the gaze based on the extracted reflected image.

Note that, if there is clearly no eyeglass reflected image present inthe image, the processing for identifying the corneal reflected imagebased on a distance between two points of the reflected images, an angleof the line connecting two points of the reflected images, or areas ofthe reflected images need not be performed. For example, if there areonly two reflected images in the image, it may be considered that thereis no eyeglass reflected image present and the two reflected images arecorneal reflected images of light of the two light sources 110 a, 110 breflected at the cornea, so processing for identifying the cornealreflected images need not be performed.

FIG. 5 is a schematic view showing the function blocks of the processor152 of the control device 150 for the above such processing. Theprocessor 152 of the control device 150 has a feature extracting part152 a, reflected image extracting part 152 b, candidate selecting part152 c, distance calculating part 152 d, angle calculating part 152 e,area calculating part 152 f, corneal reflected image identifying part152 g, threshold value correcting part 152 h, and gaze detecting part152 i. These parts which the processor 152 have are, for example,function modules realized by a computer program operating on theprocessor 152. That is, the function blocks of the processor 152 arecomprised of the processor 152 and the program (software) for making itfunction. Further, this program may be recorded in a memory 154 whichthe control device 150 is provided with or a recording medium connectedfrom the outside. Alternatively, these parts which the processor 152 hasmay be dedicated processing circuits provided in the processor 152.

The feature extracting part 152 a of the processor 152 judges if a faceof a driver appears in the image generated by the camera 120 capturingthe image of the driver and, if the face of the driver appears, extractsthe image corresponding to the face of the driver. Further, if the faceof the driver appears in the image, the feature extracting part 152 ajudges whether an eye (pupil) appears in the extracted imagecorresponding to the face of the driver. At this time, the featureextracting part 152 a extracts an image of the face of the driver fromthe image which the camera 120 generates by, for example, templatematching of a template image in which a face appears and the image whichthe camera 120 generates or by input of an image which the camera 120generates into a discriminator trained by machine learning for facedetection. Similarly, the feature extracting part 152 a extracts acontour 12 of an eye such as shown in FIG. 4 and further extracts thepupil 20 by, for example, template matching of a template image in whichan eye appears and the image which the camera 120 generates or by inputof an image which the camera 120 generates into a discriminator trainedby machine learning for detection of an eye.

If an eye of the driver appears in the image which the camera 120generates, the reflected image extracting part 152 b of the processor152 extracts the plurality of reflected images obtained by near infraredlight projected from the light sources 110 a, 110 b being reflected inthe vicinity of the eye. More specifically, the reflected imageextracting part 152 b extracts the reflected images of near infraredlight which the light source 110 a and light source 110 b project byextracting the reflected images of near infrared light in the vicinityof the pupil 20. The reflected images of near infrared light which thereflected image extracting part 152 b extracts include, in addition tothe corneal reflected image 30 a and corneal reflected image 30 b, theeyeglass reflected image 40 and eyeglass reflected image 42.

The candidate selecting part 152 c of the processor 152 selectscombinations of two reflected images becoming candidates of cornealreflected images from among the plurality of reflected images of nearinfrared light which the reflected image extracting part 152 b extracts.If there are reflected images already judged not to be corneal reflectedimages by the later explained processing among the plurality ofreflected images of near infrared light which the reflected imageextracting part 152 b extracts, the candidate selecting part 152 cexcludes the reflected images judged not to be corneal reflected imagesin order to select the combinations of two reflected images becomingcandidates of corneal reflected images.

The distance calculating part 152 d of the processor 152 calculates thedistance between two reflected images which the candidate selecting part152 c selects for each of the combinations of two reflected images. Atthis time, the distance calculating part 152 d may find the centers ofgravity of the reflected images and calculate the distance between thecenters of gravity in order to calculate the distance between the tworeflected images. Further, the distance calculating part 152 d may alsofind the number of pixels between these centers of gravity on the imagein order to calculate the distance between two reflected images.

FIG. 6 is a schematic view showing the state of the distance calculatingpart 152 d calculating the distance between two candidates of reflectedimages in the example of FIG. 4 . As shown in FIG. 6 , if the tworeflected images which the candidate selecting part 152 c selects arethe corneal reflected image 30 a and corneal reflected image 30 b, thedistance calculating part 152 d calculates the distance L1 between thecorneal reflected image 30 a and the corneal reflected image 30 b.Further, if the two reflected images which the candidate selecting part152 c selects are the corneal reflected image 30 a and the eyeglassreflected image 40, the distance calculating part 152 d calculates thedistance L2 between the corneal reflected image 30 a and the eyeglassreflected image 40. Further, if the two reflected images which thecandidate selecting part 152 c selects are the corneal reflected image30 b and the eyeglass reflected image 42, the distance calculating part152 d calculates the distance L3 between the corneal reflected image 30b and the eyeglass reflected image 42.

The angle calculating part 152 e of the processor 152 calculates anangle of a line connecting two reflected images with a line connectingthe two light sources 110 a, 110 b when viewed from the imagingdirection of the camera 120 for the combination of two reflected imageswhich the candidate selecting part 152 c selects. As a more specificexample, if the two light sources 110 a, 110 b are arranged in thehorizontal direction, the angle calculating part 152 e calculates theangle formed by the line connecting the combination of two reflectedimages which the candidate selecting part 152 c selects with thehorizontal. At this time, the distance calculating part 152 d may alsofind the centers of gravity of the reflected images and calculate theangle formed by the line connecting the centers of gravity with thehorizontal.

FIG. 7 is a schematic view showing a state of an angle calculating part152 e calculating an angle formed by a line connecting two candidates ofreflected images with the horizontal. In the example shown in FIG. 7 ,two images of the corneal reflected image 30 a and corneal reflectedimage 30 b and one eyeglass reflected image 44 appear in the image 50.Further, FIG. 8 is a view showing enlarged a positional relationshipbetween the corneal reflected image 30 a, corneal reflected image 30 b,and eyeglass reflected image 44 shown in FIG. 7 . As shown in FIG. 7 andFIG. 8 , if the two reflected images which the candidate selecting part152 c selects are the corneal reflected image 30 a and the cornealreflected image 30 b, the angle θ1 which the vector A having the cornealreflected image 30 b as its start point and the corneal reflected image30 a as its end point forms with the horizontal axis L is calculated.Further, if the two reflected images which the candidate selecting part152 c selects are the corneal reflected image 30 a and eyeglassreflected image 44, the angle θ2 which the vector B having the cornealreflected image 30 a as its start point and the eyeglass reflected image44 as its end point forms with the horizontal axis L is calculated. Notethat, the horizontal axis L may be corrected in accordance with theorientation of the face of the driver. For example, if the face of thedriver is tilted when the driver is looking at the screen of anavigation device in the vehicle, the slant of the horizontal axis L maybe corrected in accordance with the same.

The area calculating part 152 f of the processor 152 calculates theareas of the reflected images for the combination of two reflectedimages which the candidate selecting part 152 c selects. The areacalculating part 152 f may also for example find the numbers of pixelsof the regions of the reflected images on the screen to therebycalculate the areas of the reflected images.

The corneal reflected image identifying part 152 g of the processor 152excludes the eyeglass reflected images of light projected from the lightsources 110 a, 110 b reflected at the eyeglasses based on predeterminedfeatures so as to identify the combination of the corneal reflectedimage, to identify the corneal reflected image to be used for gazedetection. The predetermined features include the distance which thedistance calculating part 152 d calculates, the angle which the anglecalculating part 152 e calculates, or the areas which the areacalculating part 152 f calculates. The corneal reflected imageidentifying part 152 g identifies the combination of the cornealreflected images among the combinations of two reflected images becomingcandidates of the corneal reflected images which the candidate selectingpart 152 c selects based on these features.

The corneal reflected image identifying part 152 g preferably identifiesthe combination of the corneal reflected images from among the reflectedimages which the feature extracting part 152 a extracts based on all ofthese distance, angle, and area and identifies one corneal reflectedimage included in the identified combination as the corneal reflectedimage to be used for gaze detection.

On the other hand, the corneal reflected image identifying part 152 gmay also identify the combination of the corneal reflected images fromamong the reflected images which the feature extracting part 152 aextracts based on at least one of the distance, angle, and area andidentify one corneal reflected image included in the identifiedcombination as the corneal reflected image to be used for gazedetection. However, from the viewpoint of keeping an eyeglass reflectedimage from ending up being mistakenly detected as a corneal reflectedimage, the method based on distance is the highest in reliability. Thereliability falls in the order of the method based on angle and themethod based on area. Therefore, if identifying the corneal reflectedimages based on one of the distance, angle, and area, it is preferableto identify the corneal reflected images based on distance. Further, ifidentifying the corneal reflected images based on two of the distance,angle, and area, it is preferable to identify the corneal reflectedimages based on the distance and angle.

More specifically, the corneal reflected image identifying part 152 gexcludes a combination of two reflected images where the distance whichthe distance calculating part 152 d calculates exceeds a first thresholdvalue, identifies the combination of two reflected images where thedistance is the first threshold value or less as the combination ofcorneal reflected images, and identifies one reflected image included inthe identified combination as the corneal reflected image to be used forgaze detection. Further, the corneal reflected image identifying part152 g excludes a combination of two reflected images where the anglewhich the angle calculating part 152 e calculates exceeds a secondthreshold value, identifies the combination of two reflected imageswhere the angle is the second threshold value or less as the combinationof corneal reflected images, and identifies one reflected image includedin the identified combination as the corneal reflected image to be usedfor gaze detection. Further, the corneal reflected image identifyingpart 152 g excludes a combination of two reflected images where an areawhich the area calculating part 152 f calculates exceeds a thirdthreshold value, identifies the combination of two reflected imageswhere the areas are the third threshold value or less as the combinationof corneal reflected images, and identifies one reflected image includedin the identified combination as the corneal reflected image to be usedfor gaze detection.

Note that, gaze detection by the corneal reflection method is possibleby using either of the corneal reflected image 30 a of the light of thelight source 110 a reflected at the cornea and the corneal reflectedimage 30 b of the light of the light source 110 b reflected at thecornea. By determining which of the corneal reflected image 30 a andcorneal reflected image 30 b to use for gaze detection in advance, thecorneal reflected image identifying part 152 g identifies one cornealreflected image included in the identified combination of the cornealreflected images as the corneal reflected image to be used for gazedetection.

If identifying the corneal reflected images based on the distance or thearea, the threshold value correcting part 152 h of the processor 152corrects the threshold value compared with the distance or area based onthe distance from the light source 110 a and light source 110 b to theface of the driver. The distance between the corneal reflected image 30a and corneal reflected image 30 b of light of the light source 110 aand light source 110 b reflected at the cornea changes in accordancewith the distance from the light source 110 a and light source 110 b tothe face of the driver. Similarly, the area of the corneal reflectedimage 30 a of light of the light source 110 a reflected at the cornea orthe area of the corneal reflected image 30 b of light of the lightsource 110 b reflected at the cornea changes in accordance with thedistance from the light source 110 a and light source 110 b to the faceof the driver.

For this reason, by correcting the threshold value which is comparedwith the distance or area based on the distance from the light source110 a and light source 110 b to the face of the driver, the precisionwhen identifying the corneal reflected images based on the distance orarea is improved more. For example, the threshold value correcting part152 h corrects the above-mentioned first threshold value to a largervalue the longer the distance from the light source 110 a and lightsource 110 b to the face of the driver. Further, the threshold valuecorrecting part 152 h corrects the above-mentioned third threshold valueto a larger value the longer the distance from the light source 110 aand light source 110 b to the face of the driver. The threshold valuecorrecting part 152 h corrects these threshold values based on a map ortable etc. prescribing in advance the relationship between the distanceto the face of the driver and these threshold values.

At this time, the threshold value correcting part 152 h may also forexample estimate the distance from light source 110 a and light source110 b to the face of the driver based on a map or table etc. prescribingin advance the relationship between a position of a seat and thedistance to the face of the driver based on the position of the seat onwhich the driver sits. Further, if the camera 120 is comprised of astereo camera, the threshold value correcting part 152 h may alsocalculate the distance to the face of the driver from the parallax ofthe left and right images. The threshold value correcting part 152 h mayalso calculate the distance from the light source 110 a and light source110 b to the face of the driver in advance before the start ofprocessing for identifying the corneal reflected images, for example, atthe time the driver sits down.

The gaze detecting part 152 i of the processor 152 detects the gaze ofthe driver using the corneal reflected image identified by the cornealreflected image identifying part 152 g using the above-mentioned knowntechnique (corneal reflection method).

If the result of the gaze of the driver being detected by the gazedetecting part 152 i is, for example, that the gaze of the driver is notfacing the front for a predetermined time period or more, there is apossibility that the driver is driving while distracted. For thisreason, the processor 152 of the control device 150 issues aninstruction to display a warning at the display device 130. Due to this,a warning is displayed on the display device 130. Further, the processor152 of the control device 150 outputs a warning by voice from thespeaker 140. For example, a warning of “look ahead!” etc. is displayedat the display device 130 and, further, is emitted from the speaker 140.Due to this, the driver who may be driving while distracted can beprompted to turn his gaze to the front.

FIG. 9 is a flow chart showing the processing which the processor 152 ofthe control device 150 performs every predetermined control cycle.First, an image generated by the camera 120 capturing the driver isacquired (step S10). Next, the feature extracting part 152 a of theprocessor 152 judges whether the face of the driver appears in the image(step S12). If the face of the driver appears in the image, the featureextracting part 152 a judges whether the pupil of an eye of the driverappears in the image (step S14). If at step S12 the face of the driverdoes not appear in the image or if at step S14 the pupil of an eye ofthe driver does not appear in the image, the routine returns to stepS10.

If at step S14, the pupil of an eye of the driver appears in the image,it is judged whether to select a combination of candidates of thecorneal reflected images (step S16), while if selecting a combination ofcandidates of the corneal reflected images, the candidate selecting part152 c selects a combination of two reflected images becoming candidatesof the corneal reflected images (step S18).

On the other hand, as explained above, if there is clearly no eyeglassreflected image present in the image, there is no need for processingfor selecting a combination of candidates of the corneal reflected imageand identifying the corneal reflected images based on a distance betweentwo points of the reflected images, an angle of a line connecting twopoints of the reflected images, or the areas of the reflected images.Therefore, in such a case, at step S16, it is judged to not select acombination of candidates of the corneal reflected images, to proceed tostep S32, and to detect the gaze based on a predetermined cornealreflected image among two corneal reflected images.

Next, the distance calculating part 152 d calculates the distancebetween the two candidates of reflected images, and the cornealreflected image identifying part 152 g judges if the distance betweenthe candidates is the first threshold value or less (step S20). If thedistance between the candidates is the first threshold value or less,the angle calculating part 152 e calculates the angle formed by thevector between the two candidates of reflected images and the horizontalaxis, and the corneal reflected image identifying part 152 g judgeswhether the angle formed between the vector between candidates and thehorizontal axis is the second threshold value or less (step S22).

If the angle formed by the vector between candidates and the horizontalaxis is the second threshold value or less, the area calculating part152 f calculates the areas of the candidates of the two reflectedimages, and the corneal reflected image identifying part 152 g judgeswhether the areas of the candidates of the two reflected images are thethird threshold value or less (step S24). If the areas of the candidatesof the two reflected images are the third threshold value or less, it isjudged whether all of the combinations of candidates have been selected(step S26). If all of the combinations of candidates have been selected,the corneal reflected image identifying part 152 g identifies thecombination of the corneal reflected images from among the combinationsof candidates of the corneal reflected images (step S28). Specifically,the corneal reflected image identifying part 152 g identifies as thecombination of the corneal reflected images the two reflected imageswith a distance between candidates of the first threshold value or less,an angle formed by the vector between candidates and the horizontal axisof the second threshold value or less, and areas of the third thresholdvalue or less.

Next, the corneal reflected image identifying part 152 g identifies onecorneal reflected image included in the combination of the cornealreflected images identified at step S26 as the corneal reflected imageto be used for gaze detection (step S30).

For example, in the example of FIG. 8 , if the corneal reflected image30 a and corneal reflected image 30 b are selected as candidates, if thedistance between the corneal reflected image 30 a and the cornealreflected image 30 b is designated as L4, the area of the cornealreflected image 30 b is designated as S1, the area of the cornealreflected image 30 a is designated at S2, the threshold value of thedistance is designated as L, the threshold value of the angle isdesignated as θ, and the threshold value of the area is designated as S,the conditions of L4≤L, θ1≤θ, and S1≤S and S2≤S are satisfied, so thecorneal reflected image 30 a and corneal reflected image 30 b areidentified as the combination of true corneal reflected images and,further, either of the corneal reflected image 30 a and the cornealreflected image 30 b is identified as the corneal reflected image to beused in gaze detection. On the other hand, if the corneal reflectedimage 30 a and eyeglass reflected image 44 are selected as candidates,at least one of the distance, angle, and area will not satisfy thethreshold value, so the corneal reflected image 30 a and eyeglassreflected image 44 are never identified as the combination of thecorneal reflected images.

If any of the conditions of steps S20, S22, S24, and S26 does not stand,the routine returns to step S18 where the candidate selecting part 152 cselects a combination of two reflected images becoming new candidatesfor corneal reflected images, then the processing of step S20 and on isagain performed.

If the corneal reflected image to be used for gaze detection at step S30is identified, the gaze detecting part 152 i performs gaze detectionbased on the identified corneal reflected image (step S32). After stepS32, the processing at the present control cycle ends.

Note that, in the processing of FIG. 9 , the processing of judgment isperformed in the order of the judgment based on distance (step S20),judgment based on the angle (step S22), and judgment based on the areas(step S24), but the order of judgment is not limited to this. Thejudgments may be performed by an order different from FIG. 9 . Further,the processing of these judgments may also be performed simultaneously(in parallel).

As explained above, according to the present embodiment, by projectinglight from the light projecting part 110 having two light sources spacedapart from each other toward an eye of the driver, two corneal reflectedimages are generated for one eye. Further, by excluding eyeglassreflected images based on the distance between two points of thereflected images, the angle of a line connecting two points of thereflected images, and the areas of the reflected images, eyeglassreflected images are kept from being mistakenly detected as cornealreflected images. Due to this, even in the case of an eyeglass wearer,more accurate gaze detection becomes possible.

Above, suitable embodiments according to the present invention wereexplained, but the present invention is not limited to these embodimentsand can be variously corrected and changed within the language of theclaims.

1. A corneal reflected image identifying device comprising a lightprojecting part for projecting light from two light sources which arespaced apart from each other toward an eye, an imaging part forcapturing the eye on which light is projected, and a processorconfigured to: calculate a distance between two reflected images foreach combination of two reflected images among a plurality of reflectedimages obtained by light projected from the light sources beingreflected in a vicinity of the eye by using the captured image; andexclude a reflected image of light projected from a light source beingreflected by an object covering the eye based on predetermined featuresand identify a corneal reflected image to be used for gaze detection;wherein the predetermined features include the distance which theprocessor has calculated.
 2. The corneal reflected image identifyingdevice according to claim 1, wherein the processor is further configuredto exclude a combination of two reflected images where the distanceexceeds a first threshold value and to identify one reflected imageincluded in a combination of two reflected images where the distance isthe first threshold value or less as the corneal reflected image to beused for gaze detection.
 3. The corneal reflected image identifyingdevice according to claim 2, wherein the processor is further configuredto: calculate an angle of a line connecting two reflected images with aline connecting the two light sources when viewed from the imagingdirection of the imaging part for each combination of two reflectedimages, the predetermined feature includes the angle; and exclude acombination of two reflected images where the angle exceeds a secondthreshold value and identify one reflected image included in acombination of two reflected images where the angle is the secondthreshold value or less as the corneal reflected image to be used forgaze detection.
 4. The corneal reflected image identifying deviceaccording to claim 2, wherein the processor is further configured to:calculate an area of a reflected image for each combination of tworeflected images, the predetermined feature includes the area; andexclude a combination of two reflected images where an area exceeds athird threshold value and identify one reflected image included in acombination of two reflected images where the areas are the thirdthreshold value or less as the corneal reflected image to be used forgaze detection.
 5. The corneal reflected image identifying deviceaccording to claim 2, wherein the processor is further configured tocorrect the first threshold value to a larger value the longer adistance from the light sources to the face.
 6. The corneal reflectedimage identifying device according to claim 4, wherein the processor isfurther configured to correct the third threshold value to a largervalue the longer a distance from the light sources to the face.
 7. Agaze detection device comprising a processor configured to detect a gazeby a corneal reflection method based on a corneal reflected imageidentified by a corneal reflected image identifying device according toclaim 1.